OTN transport over a leaf/spine packet network

ABSTRACT

A network element (16) includes ingress optics (22) configured to receive a client signal; egress optics (30) configured to transmit packets over one or more Ethernet links (20) in a network (12); circuitry (26, 28) interconnecting the ingress optics (22) and the egress optics (30), wherein the circuitry is configured to segment an Optical Transport Network (OTN) signal from the client signal into one or more flows; and provide the one or more flows to the egress optics for transmission over the one or more of Ethernet links (20) to a second network element (18) that is configured to provide the one or more flows into the OTN signal.

FIELD OF THE DISCLOSURE

The present disclosure generally relates to networking. Moreparticularly, the present disclosure relates to systems and methods forOptical Transport Network (OTN) transport over a leaf/spine packetnetwork.

BACKGROUND OF THE DISCLOSURE

A leaf/spine network utilizes Equal-Cost Multi-Path (ECMP) routing tospread all flows across multiple network paths. This is typically doneby hashing packet headers to choose an output link. The hash is notaware of the bandwidth of a flow (i.e., it does not differentiatebetween mice and elephant flows), so it distributes flows evenly acrosslinks the way Link Aggregation Group (LAG) would. As implied by thenames, an elephant flow is an extremely large continuous flow over anetwork link while a mice flow is a small size flow. A hash might spreadflows evenly across many paths, but it can put two elephant flows on onelink thereby creating a hotspot of congestion, which leads to packetdrops and requires a management system to re-balance the whole network.But even after re-balancing, the elephant flows can lead to poor overallutilization of available path bandwidth. It is therefore desirable toavoid the creation of elephant flows.

OTN transport (i.e., Time Division Multiplexing (TDM)) is typicallyperformed over dedicated OTN networks. In the context of transportingOTN over a packet network, i.e., encapsulation, the conventionalapproach includes OTN Over Packet Fabric Protocol (OFP) which is anImplementation Agreement from the OIF (IA #OIF-OFP-01.0), November 2011,the contents of which are incorporated by reference herein. Also,pseudo-wires can also be used. OFP and pseudo-wires provide circuitemulation. However, these conventional approaches encapsulate an entireOptical Data Unit 4 (ODU4) into a single elephant packet flow whichcauses the issues described above.

BRIEF SUMMARY OF THE DISCLOSURE

In an embodiment, a network element includes ingress optics configuredto receive a client signal; egress optics configured to transmit packetsover one or more Ethernet links in a network; and circuitryinterconnecting the ingress optics and the egress optics, wherein thecircuitry is configured to segment an Optical Transport Network (OTN)signal from the client signal into one or more flows, and provide theone or more flows to the egress optics for transmission over the one ormore of Ethernet links to a second network element that is configured toprovide the one or more flows into the OTN signal. The one or more flowscan be a plurality of mice flows each having a sequence number forreordering at the second network element, and wherein the one or moreEthernet links are a plurality of Ethernet links, such that theplurality of mice flows are sent over the plurality of Ethernet links.Packets in the plurality of mice flows can have jumbo packet sizes thatare at least 2048B. Each of the plurality of mice flows can have a largesequence number greater than 2 bits. The circuitry can be furtherconfigured to utilize Equal-Cost Multi-Path (ECMP) or variants thereofto spread each of the plurality of mice flows over corresponding links.The plurality of Ethernet links can be any of Nx10GE, Nx25GE, Nx50GE,and Nx100GE, N is an integer greater than 1, and wherein the OTN signalis an Optical Data Unit 4 (ODU4). The circuitry can be furtherconfigured to utilize the plurality of mice flows to measurelatency/congestion performance of all paths in the network, and providethe latency/congestion performance to any of a management system, anorchestrator, and a Software Defined Networking (SDN) controller. Theone or more flows can be an elephant flow having idle time intervalsartificially inserted therein. The circuitry can be further configuredto switch the elephant flow to another link based on congestion whenthere is a gap in the elephant flow greater than a latency differencebetween a current path and a new path.

In another embodiment, a method includes, in a network element havingingress optics, egress optics, and circuitry interconnecting the ingressoptics and the egress optics, receiving a client signal via the ingressoptics; segmenting an Optical Transport Network (OTN) signal from theclient signal into a one or more flows via the circuitry; providing theone or more flows to the egress optics; and transmitting the one or moreflows over the one or more of Ethernet links to a second network elementthat is configured to provide the one or more flows into the OTN signal.The one or more flows can be a plurality of mice flows each having asequence number for reordering at the second network element, andwherein the one or more Ethernet links are a plurality of Ethernetlinks, such that the plurality of mice flows are sent over the pluralityof Ethernet links. Packets in the plurality of mice flows can have jumbopacket sizes that are at least 2048B. Each of the plurality of miceflows can have a large sequence number greater than 2-bits. The methodcan further include utilizing Equal-Cost Multi-Path (ECMP) or variantsthereof to spread each of the plurality of mice flows over correspondinglinks. The method can further include utilizing the plurality of miceflows to measure latency/congestion performance of all paths in thenetwork; and providing the latency/congestion performance to any of amanagement system, an orchestrator, and a Software Defined Networking(SDN) controller. The one or more flows can be an elephant flow havingidle time intervals artificially inserted therein. The method canfurther include switching the elephant flow to another link based oncongestion when there is a gap in the elephant flow greater than alatency difference between a current path and a new path.

In a further embodiment, an apparatus includes circuitry configured toreceive a client signal via ingress optics; circuitry configured tosegment an Optical Transport Network (OTN) signal from the client signalinto a one or more flows via the circuitry; circuitry configured toprovide the one or more flows to egress optics; and circuitry configuredto cause transmission of the one or more flows over the one or moreEthernet links to a second network element that is configured to providethe one or more flows into the OTN signal. The one or more flows can bea plurality of mice flows each having a sequence number for reorderingat the second network element, and wherein the one or more Ethernetlinks are a plurality of Ethernet links, such that the plurality of miceflows are sent over the plurality of Ethernet links. The one or moreflows can be an elephant flow having idle time intervals artificiallyinserted therein.

BRIEF DESCRIPTION OF THE DRAWINGS

The present disclosure is illustrated and described herein withreference to the various drawings, in which like reference numbers areused to denote like system components/method steps, as appropriate, andin which:

FIG. 1 is a block diagram of a system illustrating a conventionalapproach to support OTN over a packet/spine network;

FIG. 2 is a block diagram of a system illustrating the presentdisclosure to support OTN over the packet/spine network overcominglimitations with existing chipsets and encapsulation techniques;

FIG. 3 is a flowchart of a process for transporting OTN signals over thepacket/spine network;

FIG. 4 is a block diagram illustrating a TDM service that is segmentedinto a plurality of Circuit Emulation (CEM) mice flows; and

FIG. 5 is a block diagram illustrating a TDM service over a CEM elephantflow with artificially inserted idle time intervals.

DETAILED DESCRIPTION OF THE DISCLOSURE

The present disclosure relates to systems and methods for OpticalTransport Network (OTN) transport over a leaf/spine packet network.Specifically, the systems and methods relate to carrying OTN signalsover the leaf/spine packet network where the leaf/spine packet networkis not confined to a single data center but distributed over ametropolitan area. The systems and methods overcome limitations withexisting chipsets and encapsulation techniques. The present disclosureincludes adaptation of OTN to the specific needs of a leaf/spine packetnetwork by segmenting into mice flows at the source, creating gaps inelephant flows that enables some spine switches to change the elephantflow's path, and utilizing a sequence number per flow in addition to asequence number per packet for the purpose of reducing size of both thesequence numbers; per-flow latency adjustment that relies in in-orderpacket delivery within the flow. The present disclosure includesproviding mice flows with jumbo packets where the packet size dependenton link speeds. The present disclosure also includes utilizing TDMcircuit emulation mice flows to measure the latency on a large number ofpaths for feedback to the management system for path re-balancingoperations. A use case of the systems and methods include ODU4 transportover Nx10GE, Nx25GE, Nx50GE, Nx100GE.

For reference, the following definitions are used herein:

Flow A sequence of packets that all share the same address headers andare the logical equivalent to a call or connection Flowlet A burst ofpackets that is separated from other bursts within the same flow by anidle time interval that is greater than a pre-defined timeout valueLong-lived flow A flow that exists for a period of time that issufficient for a management system to react to the flow in real time.For a Software Defined Networking (SDN) system, this might be on theorder of minutes or longer. For more local node management systems thismight be on the order of seconds or longer Mice flow A low-bandwidthpacket flow Elephant flow A high-bandwidth packet flow TDM service Aconstant bitrate service such as OTN, Synchronous Optical Network(SONET), etc. CEM Circuit Emulation providing the TDM service over apacket network CEM mice flow A low-bandwidth sequence of packets thatincludes a CEM header and a payload area that is a segment of the TDMservice CEM Elephant A high-bandwidth sequence of packets Flow thatincludes a CEM header and a payload area that is a segment of the TDMservice

In a first example embodiment, the present disclosure includes breakingan OTN signal, such as a 100 G ODU4 TDM service, into multiple CEM miceflows, such as 10 10 G flows. Each CEM mice flow has a sequence number,so reordering between the flows is possible (not reordering of packetswithin a single mice flow).

In a second example embodiment, an OTN signal, such as a 100 G ODU4 TDMservice, can be carried by a single 100 G CEM elephant flow that hasidle time intervals artificially inserted to create flowlet bursts, suchas of 12 Mbps each, for example. The 12 Mb burst would lead to an idletime interval roughly once per second at 100 G rates. That would givethe network a chance to re-route the elephant flow roughly once persecond. This may only need to be 10 seconds or 1 minute depending on howoften the network needs re-balancing. The length of the idle timeinterval needs to be larger than the difference in delay between the oldpath and the new path (including fiber latency, buffer latency, switchlatency, etc.). This will typically be in the microsecond range. Sothat's roughly 100 kbits of buffering @ 100 G every time an artificialidle time interval is inserted.

The advantage of the second example embodiment is that it uses lessaddress space since there is only one flow. But it requires moreoverspeed bandwidth and relies on special spine node silicon to be ableto detect flowlets. The nodes that detect the flowlets must also haveknowledge of the end-to-end path delays through the network.

FIG. 1 is a block diagram of a system 10 illustrating a conventionalapproach to support a TDM service such as OTN over a packet/spinenetwork 12. For illustration purposes, the system 10 is shown with fourline cards 14-1, 14-2, 14-3, 14-4 with the line cards 14-1, 14-2 part ofa network element 16 at a first site and the line cards 14-3, 14-4 partof a network element 18 at a second site. The network elements 16, 18are interconnected by the packet/spine network 12 which can include aplurality of Ethernet links 20. As described herein, the packet/spinenetwork 12 is a packet network with the multiple Ethernet links 20, notin a single data center, but in a metro or regional deployment, i.e.,there are the multiple Ethernet links 20 between the network elements16, 18. The multiple Ethernet links 20 can be Nx10GE, Nx25GE, Nx50GE,Nx100GE, etc. As described herein, a single elephant flow for an ODU4 orsimilar high-rate OTN signal can overwhelm a single link of the multipleEthernet links 20. Also, for illustration purposes, FIG. 1 isillustrated in a unidirectional manner from the line card 14-1 as aningress module to the line card 14-4 as an egress module. The line cards14-1, 14-4 have similar components as does the line cards 14-2, 14-3.

The line card 14-1 includes client optics 22 configured to receive aclient signal, and OTN circuitry 24 configured to map the client signalinto an OTN frame, e.g., an ODU4, and to utilize OFP encapsulation tosegment the ODU4 into a single packet flow. In an embodiment, the OTNcircuitry 24 is the DIGI OTN processor available from Microsemi. The OTNcircuitry 24 connects to interface circuitry 26 via any of 5×100GElinks, a single 400GE link, or an Interlaken (ILKN) link. In anembodiment, the interface circuitry 26 is the Jericho2c switch fromBroadcom which is a packet processor, traffic manager, and fabricinterface.

The interface circuitry 26 connects to fabric circuitry 28 in thenetwork element 16 which connects to interface circuitry 26 on the linecard 14-2. In an embodiment, the fabric circuitry 28 can be a FabricElement (FE) from Broadcom. The interface circuitry 26 on the line card14-2 can connect to a CFP2-DCO 30 (C Form-factor Pluggable-DigitalCoherent Optics) via 4×CAUI links (100 Gigabit Attachment UnitInterface), i.e., a standard Ethernet interface.

The CFP2-DCO 30 optically connects to the packet/spine network 12 whichcan include various spine switches (omitted for illustration purposes)and which ultimately connects to a CFP2-DCO 30 in the line card 14-3.Note, the line cards 14-3, 14-4 include the same components as the linecards 14-1, 14-2 in a reverse direction here in FIG. 1 based on theunidirectional flow. There is a grandmaster clock 32 that can runSynchronous Ethernet (SyncE) and/or Precision Time Protocol (PTP) andcommunicate with timing circuitry 34 in each of the line cards 14-1,14-4.

The system 10 provides OTN encapsulation (e.g., an ODU4) over thepacket/spine network 12. The OTN circuitry 24 uses OFP encapsulation tosegment ODU4 into a single packet flow with the following properties.First, all packets in the flow are limited to the same Media AccessControl (MAC) Source Address (SA)/Destination Address (DA)/Qtag, therebylimiting the generation of flows to elephant flows. An ODU4 packet flowis limited to 256-512B packets; cannot support 2048B packets to reduceoverhead. If using 400GE to connect the OTN circuitry 24 to theinterface circuitry 26, the OTN circuitry 24 is limited to 3×ODU4 of OTNbandwidth (if there are no ODU2s for example). If there is a mixture ofODU3s and ODU2s, the OTN circuitry 24 can fill 400GE. If using 5×100GEto connect the OTN circuitry 24 to the interface circuitry 24, the OTNcircuitry 24 cannot support ODU4 since this requires at least one linkgreater than 100 G to the interface circuitry 26 (e.g., requires a 400GElink even if only carrying a single ODU4). If using ILKN to connect theOTN circuitry 24 to the interface circuitry 26, the OTN circuitry 24 cansupport 5×ODU4; a single ODU4 can be segmented into multiple ILKNchannels, but the Layer 2 header is the same. The OFP flow sequencenumber is limited to 2-bits, so no ability to reassemble multiple flowswith different latencies. The interface circuitry 24 cannot perform thisreassembly function either.

The interface circuitry 26 and the spine nodes in the packet/spinenetwork 12 use ECMP flow spreading over Nx100GE metro network links 20.The spine nodes use a hash on packet fields to spread flows. The hashtreats elephant flows the same as mice flows, so elephant flows mightget grouped into a single link and cause unbalanced hot spots in thenetwork.

Therefore, it is desirable for OTN to be transported over mice flowswith large 2048B packets. So, a single TDM service such as an ODU4 wouldbe segmented into many 1-10 G CEM mice flows and spread evenly overNx100GE spine links. Reassembly would allow for inter-CEM mice flowlatency variation of 50 μs. Large packet size reduces overheadbandwidth. Note, at 100GE link rates, a 2048B packet causes 163 ns ofHead of Line (HOL) blocking, which is acceptable.

FIG. 2 is a block diagram of a system 50 illustrating the presentdisclosure to support OTN over the packet/spine network 12 overcominglimitations with existing chipsets and encapsulation techniques. Thesystem 50 replaces the OTN circuitry 24 with a Field Programmable GateArray (FPGA) 52 with an ILKN link between the FPGA 52 and the interfacecircuitry 26. Also, the system 50 can include a line card 14-5 thatincludes the OTN circuitry 24, the interface circuitry 26, and theclient optics 22.

Operation of the system 50 is described from the right (the line card14-1) to the left (the line card 14-4). At the line card 14-1, local OTNflows not bound for the packet/spine network 12 can be OFP encapsulatedvia the line card 14-5. For OTN traffic bound for the packet/spinenetwork 12, the FPGA 52 is configured to segment the OTN traffic (e.g.,a TDM service such as an ODU4) into multiple CEM mice flows and insertlarge sequence numbers in the circuit emulation fields. Each CEM miceflow has a unique L2 header. The line card 14-2 handles ECMP flowspreading and spine-compatible MAC-in-MAC or MAC-in-IP encapsulation bythe interface circuitry 26.

The line card 14-3 terminates the spine encapsulation and performs L2forwarding to the line card 14-4 by the interface circuitry 26. The linecard 14-4 reassemble flows into the TDM service. An external memory canbe used to reassemble flows with up to 150 μs latency variation.

FIG. 3 is a flowchart of a process 100 for transporting OTN signals overthe packet/spine network 12. For illustration purposes, the process 100and the other examples described herein use a TDM service as an ODU4which has a rate of about 104.8 Gbps. Those of ordinary skill in the artwill recognize the present disclosure contemplates other OTN signalssuch as ODU3 (about 40.3 Gbps), Optical Data Unit flexible (ODUflex)(variable rates), Optical Data Unit Cn (ODUCn) where C=100 Gbps and n isan integer, and the like.

The process 100 includes, in a network element having ingress optics,egress optics, and circuitry interconnecting the ingress optics and theegress optics, receiving a client signal via the ingress optics (step102). In an example, the TDM service can be an OTN signal including anODU3, an ODU4, an ODUflex with sufficient rate (e.g., >40 Gbps), anODUCn, and the like. In an embodiment, the OTN signal is preferably anODU4, ODUflex having a rate comparable to an ODU4 or higher, and anODUCn. However, the process 100 also contemplated ODU0, ODU1, ODU2, etc.The objective of the process 100 is to segment/break down the high-rateOTN signal for transport over a packet network having multiple links.

The process 100 includes segmenting an Optical Transport Network (OTN)signal from the client signal into a one or more flows via the circuitry(step 104). Optionally, the segmenting can provide a plurality of miceflows each having a unique packet header; utilizing large sequencenumber per flow; and utilizing jumbo packet sizes. The CEM mice flowsprevent the issues caused by elephant flows, such as a single TDMservice as an ODU4 into an elephant flow that can cause congestion inthe packet/spine network 12.

The process 100 includes providing the one or more flows to the egressoptics (step 106), and transmitting the one or more flows over the oneor more of Ethernet links to a second network element that is configuredto provide the one or more flows into the OTN signal (step 108).

In an embodiment, the one or more flows are a plurality of mice flowseach having a sequence number for reordering at the second networkelement, and wherein the one or more Ethernet links are a plurality ofEthernet links, such that the plurality of mice flows are sent over theplurality of Ethernet links.

Since each mice flow can follow a different path in the packet/spinenetwork 12, one mice flow might arrive earlier/later than another miceflow, perhaps hundreds of microseconds. So, a large Sequence Number (SN)is required, e.g., greater than 2-bits, and the OIF OFP standard has aninadequate 2-bit sequence number. It might seem desirable to put the SNon each packet, but that burns unnecessary bandwidth. Instead, in anembodiment, the present disclosure proposes a better technique of an SNper flow (rather than per packet) since the latency of the flow is whatis important, not the latency of each packet within the flow. And todeal with lost packets, several repeated SN values can be repeated in aflow. The key enabler is that packet/spine network 12 delivers allpackets within a flow in order. Even if the SN is placed on each packet,the extra overhead is a small percentage of a jumbo frame, so theoverhead burden is low.

Another aspect includes coupling CEM mice flows with jumbo packets. So,every packet within the CEM mice flow should be 2048-9000B large. Thisimproves bandwidth efficiency since the payload is large relative to thepacket header. At 100GE link rates, a 2048B packet would only cause 163ns of HOL blocking, which is very small and manageable. So, a 2048Bpacket behaves like a very small cell quantum with today's extremelyfast link rates—the prior art limits OTN-over-packet to 512B, which isbandwidth wasteful.

People sometimes think that mice flow means “flows with small packets,”but this is not the right meaning. The correct meaning for mice flows is“flows with low bandwidth that can have small or large packets.” WhenOTN is encapsulated in packets, the number of packet flows and thepacket size can be chosen. For example, here are possible choices forthe encapsulation of a 100 G ODU4: 1) Segment into one hundred 1 G miceflows each with a 9 kB packet size, 2) Segment into one hundred 1 G miceflows each with a 64B packet size, or 3) Segment into a single 100 Gelephant flow of 64B packets.

As described herein, elephant flows are not great in Leaf/Spinenetworks, so the present disclosure utilizes CEM mice flows in oneaspect. The CEM mice flows can have either small packets or largepackets. The idea is that choosing large packets reduces overheadinefficiencies relative to choosing small packets. Preferably, 1 G CEMmice flows are not required; 10 G flows are perfectly fine in modernLeaf/Spine networks. As described herein, the transmitting can includeECMP flow spreading and spine-compatible MAC-in-MAC or MAC-in-IPencapsulation in the packet/spine network 12. Other variants of ECMPsuch as weight ECMP can be used.

Re-ordering may add latency, but this is addressed. For multi-path delaydifferential, if the fastest path is 0 μs and the slowest path is 10 μs,then the cost of path spreading can be a 10 μs re-assembly buffer. Thereare knobs to play with here, e.g., a given OTN flow can be restricted tosimilar-delay paths, etc. Depending on the rate, jumbo packets canintroduce latency since a full packet must be received before it can beprocessed. There are knobs here as well; a mice flow might be 10 G (doesnot have to be 1 G). Packet-only timing distribution can have morejitter than a SyncE-assisted packet timing distribution. If there areBoundary Clocks at each hop, that can help.

In addition to enabling OTN circuit emulation over a packet network, thepresent disclosure can also leverage particular properties of the OTNCEM mice flows (predictable bandwidth, following many paths, etc.) toimprove leaf/spine performance for packet-only flows. In the process100, the mice flows get reordered and reassembled into a singlehigh-speed OTN signal (e.g., ODU4) flow at the far end. The mice flowscan gather information about the network 12 that is not possible inother ways. For example, the mice flows give a good representation ofthe latency/congestion performance of all the paths in the leaf/spinemesh network 12. The re-ordering process inherently provides thislatency measurement that can be output to a management system,orchestrator, an SDN controller, etc. This can further be used to adjustthe paths taken by packet-only flows (i.e., not carrying OTN) in orderto balance bandwidth in the spine. So, the TDM traffic allows a constantprobing of the network that can improve the performance of the data-onlypackets by allowing better measurement and response. Examples of themeasurements can utilize timestamp fields, a bidirectional round-tripmeasurement, etc.

The present disclosure solves another problem: in existing approachesthat create a packet elephant flow to carry an ODU4, a single 100GE linkcannot carry the ODU4 due to the additional overhead which requires morethan 100 G of bandwidth. Existing solutions solve this by requiring anILKN interface, but that is not common in chipsets (not supported byTomahawk3, Jericho2, etc.) and it does not work over a network. Bybreaking the ODU4 into mice flows, this problem is solved, and an ODU4can be transmitted over an Nx10G, Nx25G, Nx50G, or Nx100G interface(local to the board or network).

The process 100 also works alongside 1588 PTP protocol that distributesphase/frequency timing to the endpoints for the purpose of re-creatingOTN timing.

In another embodiment, in the process 100, the one or more flows are aCEM elephant flow having idle time interval artificially insertedtherein. Some spine chipsets (e.g., Broadcom Tomahawk3) have the abilityto break up an elephant flow over multiple paths whenever there is asufficient idle time interval in the flow. So, the present disclosureoptionally allows the encapsulation of a TDM service such as an ODU4into a single CEM elephant flow with artificially inserted idle timeintervals, which the spine can take advantage of to spread the bandwidthover multiple links 20. To create an idle time interval in the CEMelephant flow, some degree of overspeed is required. This is not anissue since an ODU4 already requires slightly more bandwidth than can becarried in a single 100GE. So, the present disclosure can increase theoverspeed slightly to make up for the zero-bandwidth idle time interval.The idle time interval lengths can be configurable. Specifically, theidle time interval length needs to be greater than the maximumdifference in latency between worst-case spine paths. Since leaf/spineis typically a mesh and TDM rides at top Quality of Service (QoS), thelatency differential is in the microsecond range. This level ofbuffering is achievable.

In multi-path Ethernet networks such as the network 12, packets from asingle flow could traverse different paths and therefore arriveout-of-order. To deal with this, the standard approach is to restrict asingle flow to follow a single path—same idea as an Ethernet LinkAggregation Group (LAG). That way all packets experience the samelatency since they always follow a single path. The problem with thistechnique is that once a long-lived flow is on a path, it cannot bemoved. The inability to move it can lead to fabric inefficiencies due tothe inability to adapt as congestion changes. But there is a way to movea long-lived flow without introducing re-ordering issues. If there is agap in the flow that is greater than the latency difference between thetwo paths, then the flow can be moved to a new path during the gap andnot experience any re-ordering. Let's say the new path is 1 μs fasterthan the old path. Packets on the new path would arrive before packetson the old path. But if the gap in the long-lived flow is greater than 1μs, then the packets on the new path will not arrive earlier.

Advantageously, the process 100 provides Private Line OTN transportservice over an Ethernet Switched Network, transport over high PacketDelay Variation (PDV) Ethernet Switched networks, distribution of largerelephant flows over mice flows, support for larger packets to reduceoverhead, addition of packet re-ordering capabilities, addition of SA/DAand port-based scheduling, support for larger PDV with larger SN, andthe like.

FIG. 4 is a block diagram illustrating a TDM service that is segmentedinto a plurality of CEM mice flows. Again, the TDM service can includeOTN, SONET, etc. such as an ODU4. The TDM service is segmented, e.g.,for an ODU4, segmented from 100 G to 10 10 G segments. The segments areprovided into CEM mice flows, e.g., MF1, MF2, . . . MFN, where N>1. EachCEM mice flow has a header and a payload. The header can include aSource Address (SA), Destination Address (DA), a Virtual Local AreaNetwork (VLAN) identifier, a flow identifier, Sequence Numbers (SN) 1and 2, etc.

FIG. 5 is a block diagram illustrating a TDM service over a CEM elephantflow with artificially inserted idle time intervals. The TDM service,such as an ODU4, is provided in a CEM elephant flow with no flowlets.The present disclosure includes insertion of the idle time intervals toinclude flowlets. This enables the CEM elephant flow to be switched toother paths during an idle time interval.

It will be appreciated that some embodiments described herein mayinclude one or more generic or specialized processors (“one or moreprocessors”) such as microprocessors; Central Processing Units (CPUs);Digital Signal Processors (DSPs): customized processors such as NetworkProcessors (NPs) or Network Processing Units (NPUs), Graphics ProcessingUnits (GPUs), or the like; Field Programmable Gate Arrays (FPGAs); andthe like along with unique stored program instructions (including bothsoftware and firmware) for control thereof to implement, in conjunctionwith certain non-processor circuits, some, most, or all of the functionsof the methods and/or systems described herein. Alternatively, some orall functions may be implemented by a state machine that has no storedprogram instructions, or in one or more Application Specific IntegratedCircuits (ASICs), in which each function or some combinations of certainof the functions are implemented as custom logic or circuitry. Ofcourse, a combination of the aforementioned approaches may be used. Forsome of the embodiments described herein, a corresponding device inhardware and optionally with software, firmware, and a combinationthereof can be referred to as “circuitry configured or adapted to,”“logic configured or adapted to,” etc. perform a set of operations,steps, methods, processes, algorithms, functions, techniques, etc. ondigital and/or analog signals as described herein for the variousembodiments.

Moreover, some embodiments may include a non-transitorycomputer-readable storage medium having computer readable code storedthereon for programming a computer, server, appliance, device,processor, circuit, etc. each of which may include a processor toperform functions as described and claimed herein. Examples of suchcomputer-readable storage mediums include, but are not limited to, ahard disk, an optical storage device, a magnetic storage device, a ROM(Read Only Memory), a PROM (Programmable Read Only Memory), an EPROM(Erasable Programmable Read Only Memory), an EEPROM (ElectricallyErasable Programmable Read Only Memory), Flash memory, and the like.When stored in the non-transitory computer-readable medium, software caninclude instructions executable by a processor or device (e.g., any typeof programmable circuitry or logic) that, in response to such execution,cause a processor or the device to perform a set of operations, steps,methods, processes, algorithms, functions, techniques, etc. as describedherein for the various embodiments.

Although the present disclosure has been illustrated and describedherein with reference to preferred embodiments and specific examplesthereof, it will be readily apparent to those of ordinary skill in theart that other embodiments and examples may perform similar functionsand/or achieve like results. All such equivalent embodiments andexamples are within the spirit and scope of the present disclosure, arecontemplated thereby, and are intended to be covered by the followingclaims.

1-15. (canceled)
 16. A network element comprising: ingress optics configured to receive a client signal; egress optics configured to transmit packets over one or more Ethernet links in a network; and circuitry interconnecting the ingress optics and the egress optics, wherein the circuitry is configured to segment an Optical Transport Network (OTN) signal from the client signal into a plurality of mice flows, and provide the plurality of mice flows to the egress optics for transmission over the one or more of Ethernet links to a second network element in a network that is configured to provide the plurality of mice flows to send the OTN signal.
 17. The network element of claim 16, wherein the plurality of mice flows each have a sequence number for reordering at the second network element, and wherein the one or more Ethernet links are a plurality of Ethernet links, such that the plurality of mice flows are sent over the plurality of Ethernet links.
 18. The network element of claim 16, wherein packets in the plurality of mice flows have jumbo packet sizes that are at least 2048B.
 19. The network element of claim 16, wherein each of the plurality of mice flows has a large sequence number greater than 2 bits.
 20. The network element of claim 16, wherein the circuitry is further configured to utilize Equal-Cost Multi-Path (ECMP) or variants thereof to spread each of the plurality of mice flows over corresponding links.
 21. The network element of claim 16, wherein the plurality of Ethernet links are any of Nx10GE, Nx25GE, Nx50GE, and Nx100GE, N is an integer greater than 1, and wherein the OTN signal is an Optical Data Unit 4 (ODU4).
 22. The network element of claim 16, wherein the circuitry is further configured to utilize the plurality of mice flows to measure latency/congestion performance of one or more paths in the network, and provide the latency/congestion performance to any of a management system, an orchestrator, and a Software Defined Networking (SDN) controller.
 23. An apparatus comprising circuitry configured to segment an Optical Transport Network (OTN) signal from a client signal into a plurality of mice flows, and provide the plurality of mice flows to an egress optics for transmission over one or more of Ethernet links to a second network element in a network that is configured to provide the plurality of mice flows to send the OTN signal.
 24. The apparatus of claim 23, wherein the plurality of mice flows each have a sequence number for reordering at the second network element, and wherein the one or more Ethernet links are a plurality of Ethernet links, such that the plurality of mice flows are sent over the plurality of Ethernet links.
 25. The apparatus of claim 23, wherein packets in the plurality of mice flows have jumbo packet sizes that are at least 2048B.
 26. The apparatus of claim 23, wherein each of the plurality of mice flows has a large sequence number greater than 2 bits.
 27. The apparatus of claim 23, wherein the circuitry is further configured to utilize Equal-Cost Multi-Path (ECMP) or variants thereof to spread each of the plurality of mice flows over corresponding links.
 28. The apparatus of claim 23, wherein the plurality of Ethernet links are any of Nx10GE, Nx25GE, Nx50GE, and Nx100GE, N is an integer greater than 1, and wherein the OTN signal is an Optical Data Unit 4 (ODU4).
 29. The apparatus of claim 23, wherein the circuitry is further configured to utilize the plurality of mice flows to measure latency/congestion performance of one or more paths in the network, and provide the latency/congestion performance to any of a management system, an orchestrator, and a Software Defined Networking (SDN) controller.
 30. A method comprising: in a network element having ingress optics, egress optics, and circuitry interconnecting the ingress optics and the egress optics, receiving a client signal via the ingress optics; segmenting an Optical Transport Network (OTN) signal from the client signal into a plurality of mice flows via the circuitry; providing the one or more flows to the egress optics; and transmitting the plurality of mice flows over the one or more of Ethernet links to a second network element that is configured to provide the plurality of mice flows to send the OTN signal.
 31. The method of claim 30, wherein the plurality of mice flows each have a sequence number for reordering at the second network element, and wherein the one or more Ethernet links are a plurality of Ethernet links, such that the plurality of mice flows are sent over the plurality of Ethernet links.
 32. The method of claim 30, wherein packets in the plurality of mice flows have jumbo packet sizes that are at least 2048B.
 33. The method of claim 30, wherein each of the plurality of mice flows has a large sequence number greater than 2 bits.
 34. The method of claim 30, further comprising utilizing Equal-Cost Multi-Path (ECMP) or variants thereof to spread each of the plurality of mice flows over corresponding links.
 35. The method of claim 30, wherein the plurality of Ethernet links are any of Nx10GE, Nx25GE, Nx50GE, and Nx100GE, N is an integer greater than 1, and wherein the OTN signal is an Optical Data Unit 4 (ODU4). 